{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Document ID: D83\n",
      "Venezuela and its bank advisory committee have agreed in principle on revisions to the terms of a 21 billion dlr\n",
      "billion -> |D28:5| D35:13,22| D58:5| D63:7,10| D83:9,24,27| D89:14| D90:8| D97:23| \n",
      "bank -> |D19:22| D29:12,36| D52:22| D59:3,27| D72:7| D83:1,17,22| D88:20| D89:20| D91:24| \n",
      "agreed -> |D83:5| \n",
      "principle -> |D83:6| \n",
      "revision -> |D83:7| \n",
      "Number of unique keywords in document: 8\n",
      "Similarity score: 0.356237\n",
      "\n",
      "\n",
      "Document ID: D59\n",
      "With other major banks standing to lose even more than BankAmerica if Brazil fails to service its debt the analysts\n",
      "debt -> |D18:19| D27:8| D28:16| D29:22,27,32| D35:5| D36:18| D51:19| D57:8| D58:16| D59:13,18,23| D89:15| \n",
      "bank -> |D19:22| D29:12,36| D52:22| D59:3,27| D72:7| D83:1,17,22| D88:20| D89:20| D91:24| \n",
      "even -> |D29:15| D59:6| \n",
      "fail -> |D29:20| D59:11| \n",
      "service -> |D29:21| D59:12| \n",
      "Number of unique keywords in document: 0\n",
      "Similarity score: 0.281443\n",
      "\n",
      "\n",
      "Document ID: D29\n",
      "He noted however that any potential losses would not show up in the current quarter With other major banks standing\n",
      "losse -> |D29:4,34| D58:34| D59:25| \n",
      "debt -> |D18:19| D27:8| D28:16| D29:22,27,32| D35:5| D36:18| D51:19| D57:8| D58:16| D59:13,18,23| D89:15| \n",
      "bank -> |D19:22| D29:12,36| D52:22| D59:3,27| D72:7| D83:1,17,22| D88:20| D89:20| D91:24| \n",
      "however -> |D29:1| D58:31| \n",
      "even -> |D29:15| D59:6| \n",
      "Number of unique keywords in document: 0\n",
      "Similarity score: 0.241334\n",
      "\n",
      "\n",
      "Document ID: D16\n",
      "BankAmerica Corp is not under pressure to act quickly on its proposed equity offering and would do well to delay\n",
      "quickly -> |D16:4| D49:4| \n",
      "poor -> |D16:14| D49:14| \n",
      "proposed -> |D16:5| D33:13| D49:5| \n",
      "well -> |D16:9| D44:13| D49:9| \n",
      "performance -> |D16:15| D49:15| D65:8| \n",
      "Number of unique keywords in document: 0\n",
      "Similarity score: 0.254868\n",
      "\n",
      "\n",
      "Document ID: D49\n",
      "BankAmerica Corp is not under pressure to act quickly on its proposed equity offering and would do well to delay\n",
      "quickly -> |D16:4| D49:4| \n",
      "poor -> |D16:14| D49:14| \n",
      "proposed -> |D16:5| D33:13| D49:5| \n",
      "well -> |D16:9| D44:13| D49:9| \n",
      "performance -> |D16:15| D49:15| D65:8| \n",
      "Number of unique keywords in document: 0\n",
      "Similarity score: 0.254868\n",
      "\n",
      "\n",
      "Document ID: D18\n",
      "BankAmerica stock fell this week along with other banking issues on the news that Brazil has suspended interest payments on\n",
      "thi -> |D3:3| D18:3,24,30| D45:29| D51:3,24,30| D85:3| \n",
      "new -> |D18:10,32| D51:10,32| \n",
      "week -> |D0:3| D1:1| D3:19| D18:4,31| D27:25| D51:4,31| D57:25| \n",
      "fell -> |D18:2| D51:2| \n",
      "issue -> |D18:9| D51:9| \n",
      "Number of unique keywords in document: 0\n",
      "Similarity score: 0.215186\n",
      "\n",
      "\n",
      "Document ID: D64\n",
      "Douglas repeated an earlier company projection that thirdquarter earnings will probably be off slightly from last years 40 cts a\n",
      "repeated -> |D64:1| \n",
      "projection -> |D64:4| \n",
      "thirdquarter -> |D64:6| \n",
      "probably -> |D64:9| \n",
      "range -> |D64:16| \n",
      "Number of unique keywords in document: 5\n",
      "Similarity score: 0.312178\n",
      "\n",
      "\n",
      "Document ID: D39\n",
      "The company said the warrants are exercisable for five years at a purchase price of 125 dlrs per share Computer\n",
      "Computer -> |D38:0| D39:10,21| D41:0| D42:15| \n",
      "Terminal -> |D38:1| D39:11,22| D41:1| D42:16| \n",
      "holding -> |D39:20| \n",
      "certain -> |D39:27| \n",
      "involving -> |D39:29| \n",
      "Number of unique keywords in document: 5\n",
      "Similarity score: 0.289947\n",
      "\n",
      "\n",
      "Document ID: D37\n",
      "The company said its board of directors approved a twoforone stock split of its common shares for shareholders of record\n",
      "shareholder -> |D37:10,20| D67:43| \n",
      "board -> |D32:2| D37:2,17| D77:3| \n",
      "April -> |D37:12,23| D45:28| D70:28| D78:23| D93:11| \n",
      "twoforone -> |D37:5| \n",
      "voted -> |D37:18| \n",
      "Number of unique keywords in document: 4\n",
      "Similarity score: 0.287225\n",
      "\n",
      "\n",
      "Document ID: D67\n",
      "The company said it expects the offering to be priced at 20 dlrs per unit President Howard Dean said in\n",
      "benefit -> |D67:16,31| \n",
      "company -> |D36:0| D37:0,14| D39:0,32| D40:0| D42:0| D44:0| D48:11,13| D60:13| D64:3| D67:0,13,29,41| D75:0| D77:9| D78:11,29| \n",
      "Dean -> |D62:0| D66:3| D67:9,26| \n",
      "acquisition -> |D34:13| D44:6| D45:2| D46:9| D67:21,33| D81:32| D82:13| \n",
      "priced -> |D67:4| \n",
      "Number of unique keywords in document: 24\n",
      "Similarity score: 0.339673\n",
      "\n",
      "\n",
      "Document ID: D64\n",
      "Douglas repeated an earlier company projection that thirdquarter earnings will probably be off slightly from last years 40 cts a\n",
      "repeated -> |D64:1| \n",
      "projection -> |D64:4| \n",
      "thirdquarter -> |D64:6| \n",
      "probably -> |D64:9| \n",
      "range -> |D64:16| \n",
      "Number of unique keywords in document: 5\n",
      "Similarity score: 0.254892\n",
      "\n",
      "\n",
      "Document ID: D39\n",
      "The company said the warrants are exercisable for five years at a purchase price of 125 dlrs per share Computer\n",
      "Computer -> |D38:0| D39:10,21| D41:0| D42:15| \n",
      "Terminal -> |D38:1| D39:11,22| D41:1| D42:16| \n",
      "holding -> |D39:20| \n",
      "certain -> |D39:27| \n",
      "involving -> |D39:29| \n",
      "Number of unique keywords in document: 5\n",
      "Similarity score: 0.236741\n",
      "\n",
      "\n",
      "Document ID: D81\n",
      "Brown Forman said lower corporate tax rates and the restructuring are expected to substantially improve Brown Formans earnings and cash\n",
      "Brown -> |D77:0,12| D78:0| D79:0| D80:0| D81:0,10,27| D82:0| \n",
      "Forman -> |D77:1,13| D78:1| D79:1| D80:1| D81:1,11,28| D82:1| \n",
      "corporate -> |D81:4| \n",
      "Shearson -> |D81:17| \n",
      "Lehman -> |D81:18| \n",
      "Number of unique keywords in document: 9\n",
      "Similarity score: 0.516418\n",
      "\n",
      "\n",
      "Document ID: D77\n",
      "Brown Forman Inc said its board has approved a threefortwo stock split and a 35 pct increase in the company\n",
      "dividend -> |D77:11,24| D78:17| \n",
      "cash -> |D77:10,20| D78:16| D81:13| \n",
      "Brown -> |D77:0,12| D78:0| D79:0| D80:0| D81:0,10,27| D82:0| \n",
      "Forman -> |D77:1,13| D78:1| D79:1| D80:1| D81:1,11,28| D82:1| \n",
      "threefortwo -> |D77:5| \n",
      "Number of unique keywords in document: 5\n",
      "Similarity score: 0.400946\n",
      "\n",
      "\n",
      "Document ID: D79\n",
      "Brown Forman today reported a 37 pct increase in third quarter profits to 216 mln dlrs or 100 dlr a\n",
      "increase -> |D37:24| D39:18| D77:8| D79:4,11| \n",
      "third -> |D79:5| \n",
      "reported -> |D30:2| D62:23| D79:3| \n",
      "seven -> |D35:12| D44:10| D79:10| \n",
      "record -> |D37:11| D78:25| D79:13| \n",
      "Number of unique keywords in document: 1\n",
      "Similarity score: 0.315039\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import math,copy\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "def construct_index(pr_col):\n",
    "    forward={}\n",
    "    for docID,doc in enumerate(pr_col):\n",
    "        forward[docID]={}\n",
    "        position=0\n",
    "        doc=pr_col[docID]\n",
    "        for word in doc:\n",
    "            if word not in forward[docID].keys():\n",
    "                forward[docID][word]=[position]\n",
    "            else:\n",
    "                forward[docID][word].append(position)\n",
    "            position+=1\n",
    "            \n",
    "    inverted={}\n",
    "    for docID, wordList in forward.items():\n",
    "        for word in wordList.keys():\n",
    "            if word not in inverted.keys():\n",
    "                inverted.update({word:{docID:forward[docID][word]}})\n",
    "            elif docID not in inverted[word]:\n",
    "                inverted[word].update({docID: forward[docID][word]})\n",
    "    return forward,inverted\n",
    "\n",
    "\n",
    "def preprocess(filename):\n",
    "    doc = open(filename)\n",
    "    preprocessed_col = []\n",
    "    original_col = []\n",
    "    for s in doc:\n",
    "        s = ''.join(e for e in s if e.isalnum() or e == ' ')\n",
    "        # s.translate(s.maketrans('','',string.punctuation))\n",
    "        wordList = s.strip('\\n').split(' ')\n",
    "        wordList_org = wordList.copy()\n",
    "        wordList = [word for word in wordList if len(word) >= 4]\n",
    "        wordList_org = [word for word in wordList_org if word != '']\n",
    "\n",
    "        if wordList:\n",
    "            wordList = [word[:-1] if word.endswith('s') else word for word in wordList]\n",
    "            preprocessed_col.append(wordList)\n",
    "            original_col.append(wordList_org)\n",
    "\n",
    "    doc.close()\n",
    "    return preprocessed_col, original_col\n",
    "def weight_calcualtion(forward,inverted):\n",
    "\n",
    "    wordWeight_doc=copy.deepcopy(forward)\n",
    "    \n",
    "    fNumber=len(forward)\n",
    "    wordList=list(inverted.keys())\n",
    "    weight={}\n",
    "    tf_matrix=[]\n",
    "    \n",
    "    uniqueWord=[0]*fNumber\n",
    "    \n",
    "    for word in wordList:\n",
    "        df=0\n",
    "        tf=[0]*fNumber\n",
    "        for docID,doc in forward.items():\n",
    "            if word in doc:\n",
    "                df+=1\n",
    "                tf[docID]=len(forward[docID][word])\n",
    "        tf_matrix.append(tf)\n",
    "        weight.update({word:[df,tf]})\n",
    "\n",
    "    for idocID,doc in forward.items():\n",
    "        maxtf=max([tf[idocID] for tf in tf_matrix])\n",
    "        for word in doc:\n",
    "            if weight[word][0]==1:\n",
    "                uniqueWord[idocID]+=1\n",
    "            wordWeight_doc[idocID].update({word: weight[word][1][idocID]/maxtf*math.log2(fNumber/weight[word][0])})    \n",
    "    \n",
    "    return uniqueWord,wordWeight_doc\n",
    "                \n",
    "if __name__==\"__main__\":\n",
    "    preprocessed_col,original_col=preprocess('collection-100.txt')\n",
    "    forward, inverted=construct_index(preprocessed_col)\n",
    "    uniqueWord,wordWeight_doc=weight_calcualtion(forward, inverted)\n",
    "    #doQuery('query2.txt',forward_index, inverted_index,original_col,wordWeight_doc)\n",
    "\n",
    "    #doQuery\n",
    "    fNumber=len(forward)\n",
    "    #org_queries=open(qfilename).readlines()\n",
    "    pr_query,org_query=preprocess('query-10.txt')\n",
    "    qNumber=len(pr_query)\n",
    "    \n",
    "    wordList=list(inverted.keys())\n",
    "    dVec=np.zeros((fNumber,len(wordList)))\n",
    "    dL2norm=np.zeros(fNumber)\n",
    "        \n",
    "    for docID,doc in forward.items():\n",
    "        for wordID,word in enumerate(wordList):\n",
    "            if word in doc.keys():\n",
    "                dVec[docID][wordID]=wordWeight_doc[docID][word]\n",
    "\n",
    "    for i in range(fNumber):\n",
    "        dL2norm[i]=np.sqrt(dVec[i]@dVec[i].T)\n",
    "        \n",
    "    for query in pr_query:\n",
    "        qVec=np.zeros(len(wordList))\n",
    "        for term in query:\n",
    "            qVec[wordList.index(term)]=1\n",
    "        similarity=dVec@qVec.T/(dL2norm*np.sqrt(qVec@qVec.T))\n",
    "        \n",
    "        doc_sorted=sorted(enumerate(similarity), key=lambda x:x[1],reverse=True)\n",
    "        top3=doc_sorted[0:3]\n",
    "        \n",
    "        for docID, sim in top3:\n",
    "            print('Document ID: D%d'%docID)\n",
    "            print(' '.join(original_col[docID][0:20]))\n",
    "            word_sorted=sorted(wordWeight_doc[docID].items(), key=lambda x:x[1],reverse=True)\n",
    "            for j in range(5):\n",
    "                print('%s -> |'%word_sorted[j][0],end=\"\")\n",
    "                entry=inverted[word_sorted[j][0]].items()\n",
    "                for tmpID, pos in entry:\n",
    "                    print('D%d:%s| '%(tmpID, ','.join([str(x) for x in pos])),end=\"\")\n",
    "                print('')\n",
    "            print('Number of unique keywords in document: %d'% uniqueWord[docID])\n",
    "            print('Similarity score: %f'% sim)\n",
    "            print('\\n')\n",
    "\n",
    "\n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bank</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>stock banking</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>the company share</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>company benefit shares</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Brown Forman</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     bank\n",
       "0           stock banking\n",
       "1       the company share\n",
       "2  company benefit shares\n",
       "3            Brown Forman"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "df =  pd.read_table(\"./query-10.txt\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bank\n",
      "\n",
      "stock banking\n",
      "\n",
      "the company share\n",
      "\n",
      "company benefit shares\n",
      "\n",
      "\"Brown Forman\"\n"
     ]
    }
   ],
   "source": [
    "doc = open('query-10.txt')\n",
    "for s in doc:\n",
    "    print(s)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
